Collection of affect data from multiple mobile devices

ABSTRACT

A user interacts with various pieces of technology to perform numerous tasks and activities. Reactions can be observed and mental states inferred from these performances. Multiple devices, including mobile devices, can observe and record or transmit a user&#39;s mental state data. The mental state data collected from the multiple devices can be used to analyze the mental states of the user. The mental state data can be in the form of facial expressions, electrodermal activity, movements, or other detectable manifestations. Multiple cameras on the multiple devices can be usefully employed to collect facial data. An output can be rendered based on an analysis of the mental state data.

RELATED APPLICATIONS

This application claims the benefit of U.S. provisional patentapplications “Optimizing Media Based on Mental State Analysis” Ser. No.61/747,651, filed Dec. 31, 2012, “Collection of Affect Data fromMultiple Mobile Devices” Ser. No. 61/747,810, filed Dec. 31, 2012,“Mental State Analysis Using Heart Rate Collection Based on VideoImagery” Ser. No. 61/793,761, filed Mar. 15, 2013, “Mental State DataTagging for Data Collected from Multiple Sources” Ser. No. 61/790,461,filed Mar. 15, 2013, “Mental State Analysis Using Blink Rate” Ser. No.61/789,038, filed Mar. 15, 2013, “Mental State Well Being Monitoring”Ser. No. 61/798,731, filed Mar. 15, 2013, and “Personal EmotionalProfile Generation” Ser. No. 61/844,478, filed Jul. 10, 2013. Thisapplication is also a continuation-in-part of U.S. patent application“Mental State Analysis Using Web Services” Ser. No. 13/153,745, filedJun. 6, 2011, which claims the benefit of U.S. provisional patentapplications “Mental State Analysis Through Web Based Indexing” Ser. No.61/352,166, filed Jun. 7, 2010, “Measuring Affective Data forWeb-Enabled Applications” Ser. No. 61/388,002, filed Sep. 30, 2010,“Sharing Affect Data Across a Social Network” Ser. No. 61/414,451, filedNov. 17, 2010, “Using Affect Within a Gaming Context” Ser. No.61/439,913, filed Feb. 6, 2011, “Recommendation and Visualization ofAffect Responses to Videos” Ser. No. 61/447,089, filed Feb. 27, 2011,“Video Ranking Based on Affect” Ser. No. 61/447,464, filed Feb. 28,2011, and “Baseline Face Analysis” Ser. No. 61/467,209, filed Mar. 24,2011. This application is also a continuation-in-part of U.S. patentapplication “Sporadic Collection of Mobile Affect Data” Ser. No.14/064,136, filed Oct. 26, 2012, which claims the benefit of U.S.provisional patent applications “Sporadic Collection of Affect Data”Ser. No. 61/719,383, filed Oct. 27, 2012, “Optimizing Media Based onMental State Analysis” Ser. No. 61/747,651, filed Dec. 31, 2012,“Collection of Affect Data from Multiple Mobile Devices” Ser. No.61/747,810, filed Dec. 31, 2012, “Mental State Analysis Using Heart RateCollection Based on Video Imagery” Ser. No. 61/793,761, filed Mar. 15,2013, “Mental State Data Tagging for Data Collected from MultipleSources” Ser. No. 61/790,461, filed Mar. 15, 2013, “Mental StateAnalysis Using Blink Rate” Ser. No. 61/789,038, filed Mar. 15, 2013,“Mental State Well Being Monitoring” Ser. No. 61/798,731, filed Mar. 15,2013, and “Personal Emotional Profile Generation” Ser. No. 61/844,478,filed Jul. 10, 2013. This application is also a continuation-in-part ofU.S. patent application “Mental State Analysis Using Web Services” Ser.No. 13/153,745, filed Jun. 6, 2011 which claims the benefit of U.S.provisional patent applications “Mental State Analysis Through Web BasedIndexing” Ser. No. 61/352,166, filed Jun. 7, 2010, “Measuring AffectiveData for Web-Enabled Applications” Ser. No. 61/388,002, filed Sep. 30,2010, “Sharing Affect Data Across a Social Network” Ser. No. 61/414,451,filed Nov. 17, 2010, “Using Affect Within a Gaming Context” Ser. No.61/439,913, filed Feb. 6, 2011, “Recommendation and Visualization ofAffect Responses to Videos” Ser. No. 61/447,089, filed Feb. 27, 2011,“Video Ranking Based on Affect” Ser. No. 61/447,464, filed Feb. 28,2011, and “Baseline Face Analysis” Ser. No. 61/467,209, filed Mar. 24,2011. The foregoing applications are each hereby incorporated byreference in their entirety.

FIELD OF ART

This application relates generally to analysis of mental states and moreparticularly to analysis of mental states collected from multiplesources.

BACKGROUND

People spend an ever-increasing amount of time interacting withcomputers, and consume a vast amount of computer-delivered media. Thisinteraction may be for many different reasons, such as to obtaineducational content, to be entertained and find sources ofentertainment, to interact using social media, to create documents, andto play games, to name a few.

In some cases, the human-computer interaction may take the form of aperson performing a task using a software-based tool running on acomputer. Examples may include creating a document, editing a video,and/or doing one or more of the numerous other activities performable bya modern computer. The person may find the execution of certainactivities interesting or even exciting, and may be surprised at howeasy it is to perform the activity. The person may become excited,happy, or content as he or she performs an activity. On the other hand,the person may find some activities difficult to perform, and may becomefrustrated or even angry with the computer or software tool. In somecases, users may be surveyed in an attempt to determine whether acomputer or computer program functioned well, for example, as well as toidentify where the computer program may need improvement. However, suchsurvey results are often unreliable because the surveys are oftencompleted well after the activity was performed. In addition, surveyparticipation rates may be low, and people may not provide accurate andhonest answers to the survey.

In other cases of human-computer interaction, the person may not beusing a software tool to accomplish a task, but instead may be consumingcomputer-accessed content or media, such as news, pictures, music, orvideo. Currently, people consuming computer-driven content may tediouslyself-rate the media to communicate personal preferences. In some cases,viewers enter a specific number of stars corresponding to a level oflike or dislike, while in other cases, users are asked to answer a listof questions. While such a system of evaluation may be a helpful metricby which to evaluate media and other products or services, theevaluation may also prove tedious and challenging. Thus, in many cases,this type of subjective evaluation is neither a reliable nor practicalway to evaluate personal responses to media. Recommendations based onsuch a system of star rating and/or other means of self-reporting areimprecise, subjective, unreliable, and are further limited by samplesize, as, in past experiences, only a small number of viewers haveproven to actually rate the media they consume.

SUMMARY

Consumers interact with multiple computing devices in a variety of tasksand/or activities. In response to such an interaction, a user will reactwith a specific mental state. Such a mental state can express itself inone or more of many ways such as facial expressions, electrodermalactivity, movements, or other externally detectable manifestations.Multiple cameras and/or other monitoring devices—that, individually orcollectively, may be referred to as a sensor or sensors—can be used tocapture one or more of the externally detectable manifestations of theuser's mental state. However, there can be conditions where one or moreof the monitoring devices are not able to continually detect themanifestation. Thus, various methods, computer program products,apparatus, and systems wherein mental state data is collected bymultiple sensors, analyzed, and an output rendered based on the analysisof the mental state data are described herein. A computer-implementedmethod for mental state analysis is disclosed comprising: obtainingmental state data which is collected on an individual from multiplesources wherein the multiple sources include at least two sources offacial data; obtaining analysis of the mental state data which iscollected from the multiple sources; and rendering an output based onthe analysis of the mental state data. The mental state data frommultiple sources can be tagged. Analysis can include aggregating themental state data from the multiple sources.

In embodiments, a computer program product embodied in a non-transitorycomputer readable medium for mental state analysis comprises: code forobtaining mental state data which is collected on an individual frommultiple sources wherein the multiple sources include at least twosources of facial data; code for obtaining analysis of the mental statedata which is collected from multiple sources; and code for rendering anoutput based on the analysis of the mental state data. In someembodiments, a computer system for mental state analysis comprises: amemory which stores instructions; one or more processors coupled to thememory wherein the one or more processors, when executing theinstructions which are stored, are configured to: obtain mental statedata which is collected on an individual from multiple sources whereinthe multiple sources include at least two sources of facial data; obtainanalysis of the mental state data which is collected from multiplesources; and render an output based on the analysis of the mental statedata. In embodiments, a computer-implemented method for mental stateanalysis comprises: receiving mental state data which is collected on anindividual from multiple sources wherein the multiple sources include atleast two sources of facial data; analyzing the mental state data whichis collected from multiple sources; and providing the analysis of themental state data to a client machine. In some embodiments, acomputer-implemented method for mental state analysis comprises:receiving analysis of mental state data which is collected on anindividual from multiple sources wherein the multiple sources include atleast two sources of facial data; and rendering an output based on theanalysis of the mental state data. In embodiments, acomputer-implemented method for mental state analysis may comprise:collecting mental state data on an individual from multiple sourceswherein the multiple sources include at least two sources of facialdata; analyzing the mental state data which is collected from multiplesources; and rendering an output based on the analysis of the mentalstate data.

Various features, aspects, and advantages of various embodiments willbecome more apparent from the following further description.

BRIEF DESCRIPTION OF THE DRAWINGS

The following detailed description of certain embodiments may beunderstood by reference to the following figures wherein:

FIG. 1 is a flow diagram for mental state analysis.

FIG. 2 is a diagram showing facial data collection from multipledevices.

FIG. 3 is a timeline with information tracks relating to mental states.

FIG. 4 is diagram for sensor analysis.

FIG. 5 is a system diagram for mental state analysis.

DETAILED DESCRIPTION

As a user interacts with one or more of various computing devices, theuser's mental state can provide valuable insight into the nature of thehuman-computer interaction. The mental state of the user can includesuch emotions as frustration, confusion, disappointment, hesitation,cognitive overload, fear, exhaustion, focus, engagement, attention,boredom, exploration, confidence, trust, delight, satisfaction,excitement, happiness, contentment, or many other human emotions.Understanding a user's mental state as he or she interacts with thecomputing devices can be valuable for a variety of reasons, such asdetermining which aspects of a computer program are functioning asintended and which aspects require further improvement; determiningaspects of a computer game that are too difficult for some users or areeasy for some users; measuring the effectiveness of advertisements;determining which parts of a video most please a specific user; ordetermining a user's preferences in order to better suggest what othermedia, games, or applications the specific user may find appealing, toname a few potential applications.

While consuming media, the user can exhibit physical manifestations ofhis or her mental state, such as facial expressions, physiologicalreactions, and movements. Sensors coupled to a computer—in someembodiments, the same computer with which the user is interacting; inother embodiments, one or more other computers—are able to detect,capture, and/or measure one or more external manifestations of theuser's mental state. For example, a still camera can capture images ofthe user's face; a video camera can capture images of the user'smovements; a heart rate monitor can measure the user's heart rate; askin resistance sensor can detect changes in the user's galvanic skinresponse; and an accelerometer can measure such movements as gestures,foot tapping, or head tilts, to name a few. In embodiments, multiplesensors to capture the user's mental state data are included.

However, depending on the user and/or the sensor, certain embodimentsallow the continuous capture of all of the manifestations of mentalstates under observation. For example, when a user looks away from thecamera, certain embodiments render the capture of images of the face ofthe user impossible until they look back at the camera. As a furtherexample, a skin conductance sensor embedded in an armrest of the user'schair can only measure a galvanic skin response if the user's arm isresting on the armrest. In other cases, continuous data capture from agiven sensor is a possibility, but such capture may not be practical ordesirable due to the sheer volume of data or other factors.

Thus, as a user interacts with a cell phone, a laptop, a tablet, andother computing devices, or as images of the user are captured throughvarious other cameras, mental state data can be collected through facialimage capture. By combining the results of these variously capturedimages, the mental states of one person can be analyzed using input frommultiple sources. The combination of multiple sources can allow a morethorough coverage and capture of data, thereby making it possible toprovide higher quality mental state analysis. In some cases, images froma camera can be saved for further analysis based on pre-processing wherea user's face is detected as being visible in the image.

Once the data has been collected from the multiple devices, an analysisof the mental state data is obtained. The analysis can take place on thecomputer with which the user is interacting, the computer or computersthat captured the sensor data, and/or from one or more other computersthat are local or remote to the user. The analysis can provide themental states of the user over time based on the sensor data. Duringsome periods, data from more than one sensor is available and can beused together with data from other sensors to provide a continuousrendering of the user's mental state information. During other periods,data from one particular sensor is available and can provide continuousmental state information for the user. Further, during still otherperiods, data from another particular sensor is used to provide mentalstate information for the user. In some cases, the mental state of theuser can be estimated, interpolated, or inferred for the periods wheredata from one or more sensors was not collected.

After the analysis of the mental state data has been obtained, an outputbased on the analysis is rendered. The rendered output can include text,icons, pictures, graphs, binary data, or any other form or output that aperson or another computer can interpret, depending on the embodiment.In at least one embodiment, the rendered output includes a graph showingthe prevalence of a particular mental state over time. In someembodiments, the rendered output includes an icon that changes based onthe user's mental state. In some embodiments, the rendered outputincludes a file containing numerical data based on the obtainedanalysis. The result of the mental state analysis can also be includedin a calendar where it can be displayed or compared with the ongoingactivities already included in the calendar.

FIG. 1 is a flow diagram 100 for mental state analysis comprising acomputer-implemented method for mental state analysis. The flow 100includes obtaining mental state data 110 which is collected frommultiple sources on an individual wherein the multiple sources includeat least two sources of facial data. In embodiments, the multiplesources include multiple cameras, each positioned with a different viewof the user. In some embodiments, the multiple sources include at leastone mobile device. Any type of image capture device can be used as asource of facial data, including a webcam, a video camera, a stillcamera, a thermal imager, a CCD device, a phone camera, athree-dimensional camera, a depth camera, multiple webcams used to showdifferent views of a person, or any other type of image captureapparatus that can allow data captured to be used in an electronicsystem. In some embodiments, the facial data is collected intermittentlywhen the individual is looking in a direction of a camera, althoughthere may be times when facial data is not captured because the user'sface is not visible to any cameras. The flow 100 further comprisesperforming face detection 112 to determine whether the individual islooking towards a camera of one of the multiple devices. The flow 100further comprises filtering out faces 114 of one or more other people todetermine whether the individual is looking toward a camera of one ofthe multiple devices. By using the facial image information frommultiple devices, more comprehensive mental state data can be collected.

The flow 100 further comprises collecting other mental state data 116from the individual on a continuous basis, or, in some embodiments, onan intermittent basis. Other mental state data can include any type ofmental state data including, but not limited to, heart rate, respirationrate, blood pressure, skin resistance, audible sounds, gestures, or anyother type of data that can be useful for determining mental stateinformation. Thus in some embodiments, the other mental state dataincludes electrodermal activity data.

The flow 100 includes obtaining analysis 120 of the mental state datacollected from multiple sources. In some embodiments, obtaining analysisincludes performing the analysis on a local computer, which may be thecomputer that collected the mental state data and/or a computer withwhich a user being monitored is interacting. In some embodiments, theobtaining of analysis includes performing the analysis on a localserver, a quasi-local server—for example, a server in the same buildingor campus as the user being monitored—or on a remote server. In someembodiments, the obtaining analysis includes receiving the analysis fromanother computer, such as a local or remote server, which can be a webservice. Thus in some embodiments, the analysis of the mental state datais obtained from a web service. Because the mental state data can becollected using multiple sources, the analysis can include aggregatingthe mental state data from the multiple sources. Mental state data canbe stitched together from the various sources and the stitching mayoccur at a web service. Stitching can include using mental state data,or analysis from the mental state data, from multiple sources to providea more complete picture of the emotional experiences of the individual.In some embodiments, analysis includes identifying a best view where twoor more of the multiple sources have a camera with a view of theindividual. Mental state data from the best view may be used or givenpriority in the obtaining of the analysis. The best view, in most casesthe front view of the face, is then used in mental state analysis. Theflow 100 further comprises interpolating 122 mental state data and/ormental state analysis where the mental state data collected isintermittent. In some embodiments, the interpolating can be done betweendata sets collected from different sources. The flow may includeassembling the mental state data from the multiple sources at a webservice, analyzing the mental state data to provide mental stateinformation, and using the mental state information, based on the mentalstate data from the multiple sources to infer mental states.

The flow 100 further comprises imputing 124 additional mental state datawhere the mental state data is missing. The imputing can include fillingin blanks where data has not been collected, estimating between pointsin time where data has been collected, extrapolating from a previouslycollected data point, or the like. Analysis, based on the multiplesources of mental state data, can be used in market research. In somecases, an advertisement can be sent to an individual on a specificdevice, based on data obtained from that device and other devices. Usingthe multiple devices, a neutral mental state can be determined from onedevice and an advertisement or other media presentation can be sent tothat or another device. Further, a specific mood or emotional state canbe targeted and watched for across the various devices. When that moodis detected, an advertisement or media presentation can be sent to adesired device.

The flow further comprises partitioning the mental state data 126 basedon tagging. Tags can include various types of information includingmetadata related to the mental state data, the user being monitored, thedevice that captured the mental state data, or other types of data. Themental state data from multiple sources can be tagged with informationidentifying the source that captured the mental state data. The mentalstate data can be tagged with an identity value for the individual. Themental state data can be reassembled, based on the tagging, allowing thecombination of images collected from multiple devices. Analysis of thetagged data can allow the generation of an emotigraphic profile for anindividual. The mental state data can be tagged with information on thecontext in which the mental state data was collected. The partitioningcan separate the mental state data into two or more groups depending onthe contents of the tags.

The flow 100 further comprises inferring mental states 130 based on themental state data which was collected. Mental states that may beinferred may include one or more of a group including enjoyment,happiness, anger, sadness, stress, frustration, confusion,disappointment, hesitation, cognitive overload, focusing, being engaged,attending, boredom, exploration, confidence, trust, delight, andsatisfaction. The mental state data can include one or more of smiles,laughter, smirks, or grimaces. The mental state data can includeinformation based on the Facial Action Coding System (FACS). FACS is adetailed catalog of unique action units that correspond to independentmotions of the face. FACS enables the measurement and scoring of facialactivity in an objective, reliable, and quantitative way, and can beused to discriminate between subtle differences in facial motion.Various independent motions can be classified using action units; inembodiments, the mental state data includes FACS action units. Themental state data can include one or more of head position, up/down headmotion, side-to-side head motion, tilting head motion, body leaningmotion, or gaze direction. Various mental states can be inferred, andthe mental states can comprise one or more of a group includingfrustration, confusion, disappointment, hesitation, cognitive overload,focusing, being engaged, attending, boredom, exploration, confidence,trust, delight, or satisfaction. Once mental states are inferred, theflow 100 further comprises populating a calendar 134 based on the mentalstates which were inferred. The populating can include placing mentalstate data, mental state information, or representations of mentalstates in a timeline or calendar for a viewer's benefit or review.

The flow 100 further comprises determining contextual information 132related to the collected mental state data. Any type of contextualinformation related to the collection of the mental state data can beobtained. Some examples of collectible contextual information include atask assigned to the user, the location of the user, the environmentalconditions that the user is exposed to—such as temperature, humidity,and the like—the name of the content being viewed, the level of noiseexperienced by the user, or any other type of contextual information. Insome embodiments, the contextual information is based on one or more ofskin temperature or accelerometer data. In some embodiments, thecontextual information is based on one or more of a photograph, anemail, a text message, a phone log, or GPS information.

The flow 100 includes rendering an output 140 based on the analysis ofthe mental state data. In various embodiments, the rendering can begraphical, pictorial, textual, auditory, or any combination thereof. Therendering can include an emotigraphic profile. The rendering can bepresented on a local or remote electronic display. In some embodimentsthe rendering is printed on paper. The flow 100 further comprisesposting information based on the analysis 142 to a social network page.Various steps in the flow 100 may be changed in order, repeated,omitted, or the like without departing from the disclosed concepts.Various embodiments of the flow 100 may be included in a computerprogram product embodied in a non-transitory computer readable mediumthat includes code executable by one or more processors.

FIG. 2 is a diagram 200 showing facial data collection from multipledevices. A user 210 could be performing a task, viewing a mediapresentation on an electronic display 212, or doing something else whereit could prove useful to determine the user's mental state. Theelectronic display 212 can be on a laptop computer 220 as shown, atablet computer 250, a cell phone 240, a desktop computer monitor, atelevision, or any other type of electronic device. The mental statedata can be collected on a mobile device such as a cell phone 240, atablet computer 250, a laptop computer 220, or a watch camera 270. Thus,the multiple sources can include at least one mobile device, such as aphone 240, a tablet 250, or a wearable device such as glasses 260. Amobile device can include a forward facing camera and/or rear facingcamera that can be used to collect mental state data. The at least twosources of facial data can include one or more of a webcam 222, a phonecamera 242, a tablet camera 252, a wearable camera 262, and a roomcamera 230. A wearable camera may be some other wearable camera device.

As the user 210 is monitored, the user 210 may move due to the nature ofthe task, boredom, distractions, or for another reason. As the usermoves, the camera with a view of the user's face can change. Thus if theuser 210 is looking in a first direction, the line of sight 224 from thewebcam 222 is able to observe the individual's face, in certainembodiments, but if the user is looking in a second direction, the lineof sight 234 from the room camera 230 is able to observe theindividual's face. Further, in other embodiments, if the user is lookingin a third direction, the line of sight 244 from the phone camera 242 isable to observe the individual's face, and if the user is looking in afourth direction, the line of sight 254 from the tablet cam 252 is ableto observe the individual's face. If the user is looking in a fifthdirection, the line of sight 264 from the wearable camera 262, which maybe a device such as the glasses 260 shown and can be worn by anotheruser or an observer, is able to observe the individual's face. If theuser is looking in a sixth direction, the line of sight 274 from thewearable watch-type device 270 with a camera 272 included on the device,is able to observe the individual's face. In other embodiments, thewearable device is a another device, such as an earpiece with a camera,a helmet or hat with a camera, a clip-on camera attached to clothing, orany other type of wearable device with a camera or other sensor forcollecting mental state data. The individual 210 can also wear awearable device including a camera that is used for gathering contextualinformation and/or collecting mental state data on other users. Becausethe individual 210 can move their head, the facial data can be collectedintermittently when the individual is looking in a direction of acamera. In some cases, multiple people are included in the view from oneor more cameras, and some embodiments include filtering out faces of oneor more other people to determine whether the individual 210 is lookingtoward a camera. In some cases, multiple people may be included in theview from one or more cameras, and some embodiments include filteringout faces of one or more other people to determine whether theindividual 210 is looking toward a camera. All or some of the mentalstate data may be sporadically available from these various devices.

FIG. 3 is a timeline 310 with information tracks 300 relating to mentalstates. A first track 360 shows events that, in embodiments, are relatedto the individual's use of a computer. A first event 320 can indicate anaction that the individual took (such as launching an application); anaction initiated by the computer (such as the presentation of a dialogbox); an external event (such as a new global positioning system (GPS)coordinate); or another event such as receiving an e-mail, a phone call,a text message, or any other type of event. In some embodiments, aphotograph can be used to document an event or simply save contextualinformation in the first track 360. A second event 322 can indicateanother action or event in a similar manner. Such events can be used toprovide contextual information and can also include information such ascopies of emails, text messages, phone logs, file names, or otherinformation that can prove useful in understanding the context of auser's actions. Thus, in embodiments, contextual information is based onone or more of a photograph, an email, a text message, a phone log, orGPS information.

A second track 362 can include continuously collected mental state datasuch as electrodermal activity data 330. A third track 364 can includefacial data, which, in embodiments, is a type of mental state data thatis collected on an intermittent basis by a first camera, such as theroom cam 230 of FIG. 2 (although in some embodiments the facial data iscollected continuously). The facial data can be collected intermittentlywhen the individual is looking toward a camera. The facial data 340 caninclude one or more still photographs, videos, or abstracted facialexpressions which can be collected when the user looks in the directionof the camera. A fourth track 366 can include facial data that iscollected on an intermittent or continuous basis by a second camera,such as the mobile phone camera 242 of FIG. 2. The facial data 342 caninclude one or more still photographs, videos, or abstracted facialexpressions which can be collected when the user looks in the directionof that camera. A fifth track 368 can include facial data that iscollected from a third camera, such as the webcam 222 of FIG. 2. In theexample shown, the fifth track 368 includes first facial data 344,second facial data 346, and third facial data 348, which can be any typeof facial data including data that can be used for determining mentalstate information. Any number of samples of facial data can be collectedin any track. The mental state data from the various tracks can becollected simultaneously, collected on one track exclusive of othertracks, collected where mental state data overlaps between the tracks,and so on. When mental state data from multiple tracks overlap, onetrack's data can take precedence or the data from the multiple trackscan be combined.

Additional tracks, through the n^(th) track 370, of mental state data ofany type can be collected. The additional tracks 370 can be collected ona continuous or on an intermittent basis. The intermittent basis can beeither occasional or periodic. Analysis can further compriseinterpolating mental state data when the mental state data collected isintermittent, and/or imputing additional mental state data where themental state data is missing. One or more interpolated tracks 372 can beincluded and can be associated with mental state data that is collectedon an intermittent basis, such as the facial data of the fifth track368. Interpolated data 350 and interpolated data 352 can containinterpolations of the facial data of the fifth track 368 for the timeperiods where no facial data was collected in that track. Otherembodiments interpolate data for periods where no track includes facialdata. In other embodiments, analysis includes interpolating mental stateanalysis when the mental state data collected is intermittent.

The mental state data, such as the continuous mental state data 330and/or any of the collected facial data 340, 342, 344, 346, and 348, canbe tagged. The tags can include metadata related to the mental statedata, including, but not limited to, the device that collected themental state data; the individual from whom the mental state data wascollected; the task being performed by the individual; the media beingviewed by the individual; and the location, environmental conditions,time, date, or any other contextual information. The tags can be used tolocate pertinent mental state data; for example, the tags can be used toretrieve the mental state data from a database. The tags can be includedwith the mental state data that is sent over the internet to cloud orweb-based storage and/or services; so, the tags can be used locally onthe machine where the mental state data was collected and/or remotely ona remote server or a cloud/web service.

FIG. 4 is a diagram for sensor analysis. A system 400 can analyze datacollected from a person 410 as he or she interacts with a computer orviews a media presentation. The person 410 can have a biosensor 412attached to him or her for the purpose of collecting mental state data.The biosensor 412 can be placed on the wrist, palm, hand, head, or otherpart of the body. In some embodiments, multiple biosensors are placed onthe body in multiple locations. The biosensor 412 can include detectorsfor physiological data such as electrodermal activity, skin temperature,accelerometer readings, and the like. Other detectors for physiologicaldata can also be included, such as heart rate, blood pressure, EKG, EEG,other types of brain waves, and other physiological detectors. Thebiosensor 412 can transmit collected information to a receiver 420 usingwireless technology such as Wi-Fi, Bluetooth, 802.11, cellular, or otherbands. In other embodiments, the biosensor 412 communicates with thereceiver 420 using other methods such as a wired or optical interface.The receiver can provide the data to one or more components in thesystem 400. In some embodiments, the biosensor 412 records multipletypes of physiological information in memory for later download andanalysis. In some embodiments, the download of recorded physiologicaldata is accomplished through a USB port or other form of wired orwireless connection.

Mental states can be inferred based on physiological data, such asphysiological data from the sensor 412. Mental states can also beinferred based on facial expressions and head gestures observed by awebcam, or using a combination of data from the webcam and data from thesensor 412. The mental states can be analyzed based on arousal andvalence. Arousal can range from being highly activated—such as whensomeone is agitated—to being entirely passive—such as when someone isbored. Valence can range from being very positive—such as when someoneis happy—to being very negative—such as when someone is angry.Physiological data can include one or more of electrodermal activity(EDA), heart rate, heart rate variability, skin temperature,respiration, accelerometer readings, and other types of analysis of ahuman being. It will be understood that both here and elsewhere in thisdocument physiological information can be obtained either by biosensor412 or by facial observation via an image capturing device. Facial datacan include facial actions and head gestures used to infer mentalstates. Further, the data can include information on hand gestures orbody language and body movements such as visible fidgets. In someembodiments, these movements are captured by cameras, while in otherembodiments, these movements are captured by sensors. Facial data caninclude the tilting the head to the side, leaning forward, smiling,frowning, and many other gestures or expressions.

In some embodiments, electrodermal activity is collected, eithercontinuously, every second, four times per second, eight times persecond, 32 times per second, or on some other periodic basis. Or, insome embodiments, electrodermal activity is collected on an intermittentbasis. The electrodermal activity can be recorded and stored onto adisk, a tape, flash memory, a computer system, or streamed to a server.The electrodermal activity can be analyzed 430 to indicate arousal,excitement, boredom, or other mental states based on observed changes inskin conductance. Skin temperature can be collected and/or recorded on aperiodic basis. In turn, the skin temperature can be analyzed 432.Changes in skin temperature can indicate arousal, excitement, boredom,or other mental states. Heart rate information can be collected andrecorded, and can also be analyzed 434. A high heart rate can indicateexcitement, arousal, or other mental states. Accelerometer data can becollected and used to track one, two, or three dimensions of motion. Theaccelerometer data can be recorded. The accelerometer data can be usedto create an actigraph showing an individual's activity level over time.The accelerometer data can be analyzed 436 and can indicate a sleeppattern, a state of high activity, a state of lethargy, or other states.The various data collected by the biosensor 412 can be used along withthe facial data captured by the webcam in the analysis of mental states.Contextual information can be based on one or more of skin temperatureand/or accelerometer data. The mental state data can include one or moreof a group including physiological data, facial data, and accelerometerdata.

FIG. 5 is a system diagram for mental state analysis. The system 500 caninclude one or more computers coupled together by a communication linksuch as the Internet 510. The system 500 can also include two or morecameras that can be linked to the one or more computers and/or directlyto a communication link. The system 500 can include a mental state datacollection machine 520, which, in some embodiments, is also referred toas a client machine. The mental state data collection machine 520includes a memory 526 which stores instructions, one or more processors524 coupled to the memory, a display 522, and a webcam 528. The display522 may be any electronic display, including but not limited to, acomputer display, a laptop screen, a net-book screen, a tablet screen, acell phone display, a mobile device display, a remote with a display, atelevision, a projector, or the like. The webcam 528, as the term isused herein, may refer to a camera on a computer (such as a laptop, anet-book, a tablet, or the like), a video camera, a still camera, a cellphone camera, a mobile device camera (including, but not limited to, aforward facing camera), a thermal imager, a CCD device, athree-dimensional camera, a depth camera, and multiple webcams used tocapture different views of viewers or any other type of image captureapparatus that may allow image data captured to be used by an electronicsystem.

An individual can interact with the mental state data collection machine520, interact with another computer, or view a media presentation onanother electronic display, among other activities. The system 500 mayinclude a computer program product embodied in a non-transitory computerreadable medium including code for obtaining mental state data which iscollected on an individual from multiple sources wherein the multiplesources include at least two sources of facial data, code for obtaininganalysis of the mental state data which is collected from multiplesources, and code for rendering an output based on the analysis of themental state data. With such a program stored in memory, the one or moreprocessors 524 can be configured to obtain mental state data which iscollected on the individual from multiple sources wherein the multiplesources include at least two sources of facial data, obtain analysis ofthe mental state data which is collected from multiple sources, andrender an output based on the analysis of the mental state data. Thusthe system 500 can enable a method for collecting mental state data onan individual from multiple sources wherein the multiple sources includeat least two sources of facial data, analyzing the mental state datawhich is collected from multiple sources, and rendering an output basedon the analysis of the mental state data. The multiple sources caninclude two or more of the webcam 528, a first camera device 560 linkedthrough the internet 510, and/or a second camera device 562 linkeddirectly to the mental state data collection machine 520. In someembodiments, the mental state data collection machine 520 can sendmental state data 530 to another computer, such as the analysis server550.

Some embodiments can include a web service or analysis server 550. Theanalysis server 550 includes one or more processors 554 coupled to amemory 556 to store instructions. Some embodiments of the analysisserver 550 include a display 552. The one or more processors 554 can beconfigured to receive mental state data from the mental state datacollection machine 530, the first camera device 560, and/or othercomputers configured to collect mental state data; the mental state datacan include data from at least two sources that can be coupled to one ormore machines. The one or more processors 554 can then analyze themental state data received and provide mental state information 532. Theanalysis can produce mental state information, inferred mental states,emotigraphs, actigraphs, other textual/graphical representations, or anyother type of analysis. In some embodiments, analysis of the mentalstate data is augmented by a human coder. The analysis server 550 candisplay at least some of the analysis on the display 552 and/or canprovide the analysis of the mental state data to a client machine suchas the mental state data collection machine 520 or another clientmachine 570 to be displayed to a user. So, the system 500 can enable amethod for receiving mental state data which is collected on anindividual from multiple sources, wherein the multiple sources includeat least two sources of facial data, analyzing the mental state datawhich is collected from multiple sources, and providing the analysis ofthe mental state data to a rendering or client machine 570. In someembodiments, the analysis server 550 can be provisioned as a web serverwith the analysis of the mental state data obtained from a web service.

Some embodiments include a rendering or second client machine 570. Therendering machine 570 includes one or more processors 574 coupled tomemory 576 to store instructions, and a display 572. The client machinecan receive the analysis of the mental state data from the analysisserver 550 and can render an output to the display 572. The system 500can enable a computer-implemented method for mental state analysiscomprising receiving analysis of mental state data which is collected onan individual from multiple sources wherein the multiple sources includeat least two sources of facial data and rendering an output based on theanalysis of the mental state data.

Each of the above methods may be executed on one or more processors onone or more computer systems. Embodiments may include various forms ofdistributed computing, client/server computing, and cloud basedcomputing. Further, it will be understood that the depicted steps orboxes contained in this disclosure's flow charts are solely illustrativeand explanatory. The steps may be modified, omitted, repeated, orre-ordered without departing from the scope of this disclosure. Further,each step may contain one or more sub-steps. While the foregoingdrawings and description set forth functional aspects of the disclosedsystems, no particular implementation or arrangement of software and/orhardware should be inferred from these descriptions unless explicitlystated or otherwise clear from the context. All such arrangements ofsoftware and/or hardware are intended to fall within the scope of thisdisclosure.

The block diagrams and flowchart illustrations depict methods,apparatus, systems, and computer program products. The elements andcombinations of elements in the block diagrams and flow diagrams, showfunctions, steps, or groups of steps of the methods, apparatus, systems,computer program products and/or computer-implemented methods. Any andall such functions—generally referred to herein as a “circuit,”“module,” or “system”—may be implemented by computer programinstructions, by special-purpose hardware-based computer systems, bycombinations of special purpose hardware and computer instructions, bycombinations of general purpose hardware and computer instructions, andso on.

A programmable apparatus which executes any of the above mentionedcomputer program products or computer-implemented methods may includeone or more microprocessors, microcontrollers, embeddedmicrocontrollers, programmable digital signal processors, programmabledevices, programmable gate arrays, programmable array logic, memorydevices, application specific integrated circuits, or the like. Each maybe suitably employed or configured to process computer programinstructions, execute computer logic, store computer data, and so on.

It will be understood that a computer may include a computer programproduct from a computer-readable storage medium and that this medium maybe internal or external, removable and replaceable, or fixed. Inaddition, a computer may include a Basic Input/Output System (BIOS),firmware, an operating system, a database, or the like that may include,interface with, or support the software and hardware described herein.

Embodiments of the present invention are neither limited to conventionalcomputer applications nor the programmable apparatus that run them. Toillustrate: the embodiments of the presently claimed invention couldinclude an optical computer, quantum computer, analog computer, or thelike. A computer program may be loaded onto a computer to produce aparticular machine that may perform any and all of the depictedfunctions. This particular machine provides a means for carrying out anyand all of the depicted functions.

Any combination of one or more computer readable media may be utilizedincluding but not limited to: a non-transitory computer readable mediumfor storage; an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor computer readable storage medium or anysuitable combination of the foregoing; a portable computer diskette; ahard disk; a random access memory (RAM); a read-only memory (ROM), anerasable programmable read-only memory (EPROM, Flash, MRAM, FeRAM, orphase change memory); an optical fiber; a portable compact disc; anoptical storage device; a magnetic storage device; or any suitablecombination of the foregoing. In the context of this document, acomputer readable storage medium may be any tangible medium that cancontain or store a program for use by or in connection with aninstruction execution system, apparatus, or device.

It will be appreciated that computer program instructions may includecomputer executable code. A variety of languages for expressing computerprogram instructions may include without limitation C, C++, Java,JavaScript™, ActionScript™, assembly language, Lisp, Perl, Tcl, Python,Ruby, hardware description languages, database programming languages,functional programming languages, imperative programming languages, andso on. In embodiments, computer program instructions may be stored,compiled, or interpreted to run on a computer, a programmable dataprocessing apparatus, a heterogeneous combination of processors orprocessor architectures, and so on. Without limitation, embodiments ofthe present invention may take the form of web-based computer software,which includes client/server software, software-as-a-service,peer-to-peer software, or the like.

In embodiments, a computer may enable execution of computer programinstructions including multiple programs or threads. The multipleprograms or threads may be processed approximately simultaneously toenhance utilization of the processor and to facilitate substantiallysimultaneous functions. By way of implementation, any and all methods,program codes, program instructions, and the like described herein maybe implemented in one or more threads which may in turn spawn otherthreads, which may themselves have priorities associated with them. Insome embodiments, a computer may process these threads based on priorityor other order.

Unless explicitly stated or otherwise clear from the context, the verbs“execute” and “process” may be used interchangeably to indicate execute,process, interpret, compile, assemble, link, load, or a combination ofthe foregoing. Therefore, embodiments that execute or process computerprogram instructions, computer-executable code, or the like may act uponthe instructions or code in any and all of the ways described. Further,the method steps shown are intended to include any suitable method ofcausing one or more parties or entities to perform the steps. Theparties performing a step, or portion of a step, need not be locatedwithin a particular geographic location or country boundary. Forinstance, if an entity located within the United States causes a methodstep, or portion thereof, to be performed outside of the United Statesthen the method is considered to be performed in the United States byvirtue of the causal entity.

While the invention has been disclosed in connection with preferredembodiments shown and described in detail, various modifications andimprovements thereon will become apparent to those skilled in the art.Accordingly, the forgoing examples should not limit the spirit and scopeof the present invention; rather it should be understood in the broadestsense allowable by law.

What is claimed is:
 1. A computer-implemented method for mental stateanalysis comprising: obtaining mental state data which is collected onan individual from multiple sources wherein the multiple sources includeat least two sources of facial data; obtaining analysis of the mentalstate data which is collected from the multiple sources; and renderingan output based on the analysis of the mental state data.
 2. The methodof claim 1 wherein the mental state data from multiple sources istagged.
 3. The method of claim 2 wherein the mental state data is taggedwith an identity value for the individual.
 4. The method of claim 3wherein the mental state data is tagged with information on context inwhich the mental state data was collected.
 5. The method of claim 2further comprising partitioning the mental state data based on tagging.6. The method of claim 1 wherein the at least two sources of facial datainclude one or more of a webcam, a phone camera, a tablet camera, awearable camera, a room camera, a mobile device, a cell phone, a tabletcomputer, or a laptop computer.
 7. The method of claim 1 wherein themultiple sources include at least one mobile device.
 8. The method ofclaim 7 wherein the at least one mobile device includes a forward facingcamera.
 9. The method of claim 1 wherein the analysis of the mentalstate data is obtained from a web service.
 10. The method of claim 1wherein the analysis includes aggregating the mental state data from themultiple sources.
 11. (canceled)
 12. The method of claim 1 wherein thefacial data is collected intermittently while the individual is lookingtoward a camera.
 13. The method of claim 12 further comprisingperforming face detection to determine whether the individual is lookingtoward the camera.
 14. The method of claim 13 further comprisingfiltering out faces of one or more other people to determine whether theindividual is looking toward the camera.
 15. The method of claim 1further comprising interpolating mental state data when the mental statedata collected is intermittent.
 16. The method of claim 1 furthercomprising interpolating mental state analysis when the mental statedata collected is intermittent. 17-18. (canceled)
 19. The method ofclaim 1 further comprising imputing additional mental state data for oneor more periods where no mental state data was collected.
 20. The methodof claim 1 further comprising determining contextual information. 21.The method of claim 20 wherein the contextual information is based onone or more of skin temperature, accelerometer data, a photograph, anemail, a text message, a phone log, or GPS information.
 22. The methodof claim 1 further comprising inferring mental states based on themental state data which was collected.
 23. The method of claim 22wherein the mental states inferred include one or more of frustration,confusion, disappointment, hesitation, cognitive overload, focusing,being engaged, attending, boredom, exploration, confidence, trust,delight, or satisfaction.
 24. The method of claim 23 further comprisingpopulating a calendar based on the mental states which were inferred.25. (canceled)
 26. The method of claim 1 further comprising postinginformation based on the analysis to a social network page.
 27. Themethod of claim 1 wherein the mental state data includes one or more ofa group including physiological data, facial data, or accelerometerdata.
 28. The method of claim 27 wherein the physiological data includesone or more of electrodermal activity, heart rate, heart ratevariability, skin temperature, or respiration.
 29. A computer programproduct embodied in a non-transitory computer readable medium for mentalstate analysis, the computer program product comprising: code forobtaining mental state data which is collected on an individual frommultiple sources wherein the multiple sources include at least twosources of facial data; code for obtaining analysis of the mental statedata which is collected from multiple sources; and code for rendering anoutput based on the analysis of the mental state data.
 30. A computersystem for mental state analysis comprising: a memory which storesinstructions; one or more processors coupled to the memory wherein theone or more processors, when executing the instructions which arestored, are configured to: obtain mental state data which is collectedon an individual from multiple sources wherein the multiple sourcesinclude at least two sources of facial data; obtain analysis of themental state data which is collected from multiple sources; and renderan output based on the analysis of the mental state data. 31-33.(canceled)